Automatic frame selection and layout of one or more images and generation of images bounded by a frame

ABSTRACT

A method is disclosed of analyzing an image to identify “must exclude” regions and of determining the best fit of an image outline so as to generate an image boundary that excludes the majority of “must exclude” regions. The method is applicable to the automatic generation of greeting cards and photo-montages.

FIELD OF THE INVENTION

The present invention relates to a method of defining an image boundarywhich acts to frame an image or which serves to define an area of animage which is to be displayed, a computer program product which, whenrun on a data processor, causes the data processor to execute the methodof defining an image boundary, and an image processing system fordefining an image boundary and for presenting one or more bounded imagesfor display.

BACKGROUND OF THE INVENTION

There already exists software and image processing systems that allow auser to position a captured image within an image boundary, therebycropping (i.e. excluding portions) the image to generate greeting cardsor photo-montage layouts, or other similar products. The shape of theimage boundary is selected by the user. The selection of the shape ofthe image boundary is, in some circumstances, particularly burdensome tothe user. For example, when wishing to generate a greetings card a usermust select one of a number of different generic image boundary shapes.The appropriate selection of the shape of the image boundary for anyparticular image to generate a pleasing end result may either be timeconsuming, or beyond the artistic capabilities of the user. Additionallyor alternatively, the user may simply wish to generate the greetingscard in as a short time as possible.

SUMMARY OF THE INVENTION

It is also known to provide photo-montages. A photo-montage is a numberof individual images individually arranged within an overall boundary(which defines the extent of a display region) and/or arranged tooverlap one another. The time taken to manually select appropriateimages and to either physically or electronically “cut and paste” andscale the selected images to produce the final montage can beprohibitive to many users, even though it has been found that theresulting photo-montage is a particularly pleasing way of presenting anumber of images.

It would therefore be advantageous to provide a method of and apparatusfor automating the selection of images to create greeting cards,photo-montages, or the like and to automatically define appropriateimage boundaries for such greeting cards or photo-montages.

The main requirement in such a method and system is to define anappropriate image boundary, either with reference to a number ofpredefined image outlines, or simply in relation to the image itself.

According to a first aspect of the present invention there is provided amethod of arranging at least one image within a display area, the methodcomprising the steps of: analysing the or each image in accordance withat least one predefined composition rule; for the or each imageselecting an image boundary the shape of which, in accordance with saidanalysis, best fits said image; applying the boundary to the at leastone image to produce a processed image; and placing the at least oneprocessed image within said display area.

Preferably the analysing step comprises identifying one or more featuresof compositional significance and/or identifying a number of regions ofinterest within the image. A region of interest generally denotes somefeature within the image. A region of compositional significance can bea region of interest or a region which is not interesting, for example,background. A single region or a combination of regions of interest isthen selected and the at least one composition rule is applied to thefeatures of compositional significance that are located within andadjacent a selected combination of regions of interest. This allows animage or a sub-portion of the image to be analysed to identify regionsto be included and regions that are to be excluded The portion of theimage can be regarded as a composition map as it identifies the includedand excluded features of the image.

Preferably the shape of image boundary that optimally includes the“include” regions and optimally excludes the “exclude” regions for eachcombination of regions in the composition map is defined as the bestfit.

Preferably, one or more parameters of the shape of the image boundarymay be adjusted to optimise this best fit. Preferably the image boundaryshape is selected from a plurality of geometric shapes.

Preferably, once the processed image has been generated, it ispositioned in the display area in accordance with at least onepredetermined placement rule.

The display area may comprise a predefined arrangement of imageboundaries. Alternatively, a user may define one or more of the number,position and size of the image boundaries. The user may also control oneor more parameters of the display area so as to control the overallstyle of the display area, for example, specifying a single genericshape (rectangle, ellipse, etc.) to be used for all image boundaries inthe display area.

Images, (especially but not exclusively for greeting cards), may have aborder added whose colour and style is chosen automatically. The colourof the border may be selected on the basis of the dominant colours inthe image. Thus, for example, the border colour chosen could beautomatically selected to match a strongly salient colour in theimage—such as a red dress worn by a subject of the image.

Alternatively, the border colour could be selected so as to match thelargest area colour of the image.

Alternatively, the border colour could be chosen to deliberatelycontrast with one of the largest areas of colour of the image. Such aschoosing a dark frame for a light image.

Alternatively a look up table or some other colour mapping method couldprovide a harmonising colour for the dominant colour in the image.

Many other automatic methods of selecting the border colour by analysingthe significant colours in the image could be implemented if desired.

The style (pattern) of the border could also be selected automatically.Thus, for example, the style could be a random choice from a set ofalternatives, a choice based on the users previously recordedpreferences, or selected on the basis of the overall texture of theimage. Various measures of texture would be applicable. For example,whether the image contains a predominance of straight lines, if so, usea border style which is relatively plain and straight. If not, then usea slightly patterned border style.

Preferably the number of image boundaries placed within the display areais equal to the number of original images. However, the number of imagesand image boundaries need not be matched. Thus, for example, if thereare more images than holes available in a montage template, then someimages cannot be used and must be discarded. One basis for doing thismight be to rank images by size of the regions of interest therein andto exclude those images with only small regions of interest. In the casewhere there are more holes to fill in the montage template then eachimage is used once. Then a search for alternative crops from the imagesmay be performed. For example, regions of interest may be re-analysed todefine sub-regions to be displayed. Thus, if a group of peoplerepresented one region of interest, then a sub-region may be a close upof a face of one of the people.

It is thus possible to provide a method of arranging N images within adisplay area, where N is a positive integer greater than zero, themethod comprising the steps of:

-   i. Selecting an Ith image, where I is a positive integer in the    range 1 to N, inclusive;-   ii. Analysing the Ith image in accordance with at least one rule of    image composition to define at least one include region and at least    zero exclude regions;-   iii. Repeating the steps i and ii for all images;-   iv. Selecting for the Ith image an image boundary which, when    applied to the Ith image, includes at least one of the include    regions within the boundary and excludes any exclude regions; but    which also takes account of other ones of the images so as to    optimise the arrangement of images and image boundaries;-   v. Applying the image boundary to the Ith image to generate an Ith    processed image; and-   vi. Positioning the Ith image within the display area in accordance    with at least one positioning rule.

Such a method is of use when generating a montage where the images areplaced in a defined template.

The analysis may also identify “don't care” regions where those regionsor portions of those regions may be included within the image boundaryso as to aid placement of the include region within the boundary.

There are constraints and problems which are faced in the generation ofmontages. In the case where the arrangement of holes is predefined bythe montage template, the optimisation problem is that of allocatingimages to holes. The aspect ratio and size of the holes are fixed by thetemplate. It is then a matter of heuristic optimisation to decide whichimage should be used to fill which hole. Thus, wide thin holes may begood for landscapes, whereas small square holes may be better for faces.The invention compares images with the holes available and generates thebest crop for each possible hole/image combination and then selects thebest overall set such that all the holes are filled with reasonably goodcrops. Thus the optimisation is global across all images.

However, it is also possible that the template merely acts as a guideand that the individual holes within the template may be adjustable interms of size and aspect ratio.

Consider for example a template made up of rectangular holes. It may bepossible to allow the hole sizes to be adjusted. However, in order tomaintain the overall appearance of the template it may also be necessaryto place constraints on the size of the boundary between the holes. Thismay entail defining maximum and minimum distances between the holes (oreven defining a fixed distance). Thus changes in the size of any holemay affect the size of one or more of the other holes in the montage.

Thus the fitting problem may be global, in that all fits need to beoptimised together.

In the case of a non-overlapping montage where the system has somelatitude in generating the arrangement of holes, that is the genericshape used, the number of shapes, choice of size and aspect ratio, thenthe optimisation problem has more degrees of freedom. The problem isthen not quite as simple as cropping each image independently and thenrearranging the resulting crops in the display area. However, thisapproach could be used. A better approach is to use a global constraintto ensure that there is an appropriate range of shapes to ensure thatthe montage has an interesting variety of shapes, and also to ensurethat the shapes fit together in a pleasing manner. One way of achievingthis is to pick a subset of key images, typically less than five, andgenerate the best independent crops for these and then place them in thedisplay area such that they are large and fairly well distributed. A setof smaller crop boundaries may then be generated to fill the remainingspace in the display area. The allocation of the remaining images to theholes is then optimised in a way similar to that for the predeterminedtemplate.

In one embodiment of the invention, relating to the generation ofoverlapping montages, a plurality of image boundaries are placed withinthe display area with each image boundary overlapping or beingoverlapped by at least a portion of another image boundary to form anoverlapping montage of images. In this case, the analysing step furthercomprises identifying “don't care” regions of the composition map. The“don't care” regions are regions of or regions adjacent the selectedcombinations of regions of interest that may optionally be excluded fromthe processed image. Preferably the overlapping portions of the imagemontage comprise the “don't care” regions. The don't care regions neednot be explicitly calculated if “include” and “exclude” regions aredefined since the don't care regions are what's left.

According to a second aspect of the present invention there is provideda method of arranging a plurality of images within a display area, themethod comprising the steps of: analysing each of said plurality ofimages in accordance with at least one compositional rule to therebyidentify one or more regions of each image to be excluded from saidimage; placing said analysed images within said display area, such thatsome images overlap a portion of at least one other image, whereby saidoverlapping portions obscure one or more of said regions to be excluded.

Preferably the regions to be excluded from images that are notthemselves obscured by overlapping portions are cropped from the image.Preferably the analysing step of this second aspect of the presentinvention is the same as that as for the first aspect of the presentinvention and identifies the regions to be excluded from the images.

Preferably the analysis also identifies “don't care” regions which maybe partially or wholly overlapped by another image.

Preferably the placement of the overlapping images within the displayarea is determined by one or more parameters of the images. Suchparameters may include the composition of the image, the overallbrightness of the image or the contrast between regions of interest andremaining portions of the image. These parameters may be adjustable by auser.

According to a third aspect of the present invention there is providedan image processing apparatus for arranging a number of images within adisplay area, comprising: an input arranged to receive said images; animage processor coupled to said input and arranged to arrange saidimages in accordance with the method according to the first or secondaspect of the invention; and an output coupled to said image processorarranged to output said arrangement of images within the display area.

According to a fourth aspect of the present invention the imageprocessing apparatus of the third aspect of the present invention isprovided in combination with an image capture device, wherein the inputof the image processing system is arranged to receive the capturedimages from the image capture device.

Preferably the output of the image processing system is coupled to animage storage device. The image storage device may be a separatecomputer, computer hard disk, or any other suitable storage medium.

Preferably the output of the image processing system is coupled to aprinter, thereby allowing hard copies of the arrangement of the imagesto be printed out.

BRIEF DESCRIPTION OF THE DRAWINGS

According to a fifth aspect of the present invention there is provided acomputer program product comprising computer readable instructions forcausing a computer to execute the method of the first or second aspectsof the present invention.

The present invention will now be described, by way of example, withreference to the accompanying figures, in which:

FIG. 1 schematically illustrates the application of an embodiment of thepresent invention to generate a greetings card;

FIG. 2 schematically illustrates a finished greeting card.

FIG. 3 schematically illustrates the application of an embodiment of thepresent invention to the generation of a photo-montage;

FIG. 4 schematically illustrates an embodiment of the present inventionto generate a free form photo-montage;

FIG. 5 shows an image to which embodiments of the present invention maybe applied;

FIG. 6 shows the image of FIG. 5 having areas of averaged colour andintensity merged together;

FIG. 7 shows a compositional significance map of the image shown in FIG.5; and

FIG. 8 schematically illustrates an image processing system constitutingan embodiment of the present invention.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates the operation of an embodiment of thepresent invention in relation to the generation of a greeting card orthe like. The user presents an image 1 that is to be the subject of thegreeting card. The image may be downloaded from a camera, or selectedfrom a user's “album” of images or selected from a pre-prepared store ofimages. In FIG. 1 the image 1 has four distinct areas of compositionalsignificance, namely: the sky 2, the building 3 substantially in thecentre of the image, the ground 4 and the partial tree 5 on the far lefthand side of the image. The image 1 shown is a particular example wherethe composition of the image may be improved by removing the partialarea of significance, in this case the tree 5, from the very edge of theimage. An area, or feature, of compositional significance is any area orfeature of the image that may be used by a compositional rule. Suchfeatures of significance may include regions of interest, horizonpositions, significantly contrasting boundaries between regions of theimage, areas of intense colour or brightness, areas of low intensity,brightness or contrast (i.e. probably boring areas) and “busy” regionsof the image. A region of interest is an area of an image thatconstitutes a subject. An image may include more than one region ofinterest.

The image is analysed (as described hereinafter) to identify regions ofinterest and the areas of compositional significance and to apply one ormore rules of composition to the image on the basis of the identifiedareas. Such a rule of composition may be the rule of thirds, where theimage is sub-divided by two parallel horizontal and two parallelvertical lines in a manner of a noughts and crosses board and areas ofsignificance that are be included are positioned on the intersections,or close to the intersections, of the horizontal and vertical lines.Another possible rule of composition would be to remove any areas ofsignificance located at the very edge of the total image.

After performing this analysis one or more composition maps aregenerated for the image. Each composition map defines a combination of“include” regions of interest and “exclude” regions of significance. The“include” and “exclude” regions are determined by applying the selectedrules of composition in conjunction with the features of compositionalsignificance appropriate to the particular combination of regions ofsignificance. It will be appreciated that a more refined allocationsystem having a number of different “rankings” may be applied.

A plurality of predefined, generic shapes that are provided as imageboundaries 6-11 are shown. These include a square 6, a rectangle 7, anellipse 8, an circle 9, a star 10 and a triangle 11. This list is notexhaustive and more complex, and indeed non-geometric, shapes may alsoprovided.

A matching process is applied to each of the generic image boundaries 6to 11 so as to determine the best fit of region of interest combinationwith image boundary. The best fit is the pairing of generic imageboundary and the composition map that includes as many of the “include”regions of the image as possible, whilst excluding as many of the“exclude” regions as possible. That is to say, the best fit is providedby that shape of image boundary that provides an image that bestconforms to the selected rules of composition. The best fit isgenerally, but not necessarily, the generic shape of image boundary thatnot only excludes the “exclude” regions and includes the “include”regions, but also minimises large boring areas of the image. The bestfit may be determined by applying a set of heuristic rules to thebounded images that generate a “penalty” for the bounded image. Thepenalties generated by each rule may be combined as a weighted sum forthe image, and the weighted sum for each image/boundary combination usedto determine the best fits. One such method is described in theapplicant's co-pending UK application, GB0031423.7. It will beappreciated by those skilled in the art that other known methods ofimage evaluation may be used instead of, or in addition to, this method.Parameters of the shape of the image boundaries, including the aspectratio, may be varied to best accomplish this. The image boundary shapeparameters will vary depending on the generic shape. For example, forthe rectangle 7 the aspect ratio may be varied, yet for the star 10 thenumber of arms and their length relative to the star size may be varied.

The best fit of the predefined image boundaries 6 to 11 is selected asthe final image boundary. The boundary is then used to crop the image soas to create a processed image. The processed image is then applied tothe available display area, which in the case of a greetings card istypically the rectangular front of the card. The display area layouti.e. the arrangement of the or each processed image on the display areacan also be varied. The position and size of each processed image asdefined by its image boundary on the display area may be varied. If animage boundary has already been matched to a composition map for animage then adjustment of the size of the image boundary on the displayarea causes a corresponding scaling of the image within the imageboundary.

Thus, supposing that the automatic analysis had been applied to theimage 1 of FIG. 1, the compositional rules would identify the regions ofcompositional significance 2, 3, 4 and 5. Of these, the regions 3 and 5,that is the castle and partial tree would be identified as regions ifinterest, whereas regions 2 and 4, that is the sly and ground, would beidentified as boring areas. The analysis then identifies the areas asfollows:

REGION STATUS 2 Don't care 3 Include 4 Don't care 5 Exclude

The algorithm may then seek to apply a boundary from the generic shapes,based on the image analysis and the shape of a target display area. Inthis case the target display area is defined by the perimeter 12 of thegreeting card.

The fitting process may scale or rotate the generic shapes. Thus, forexample, a quarter rotation of ellipse 8 to generate an image boundary13 framing the castle 3 and excluding the tree 5 may be presented to theuser as a finished card, as shown in FIG. 2.

The colour and texture/pattern of the border 12 a of the finished cardmay also be selected automatically, or semi-automatically. Thus thecompositional rules could be used to identify the dominant orsignificant colours in the image. The border may then be selected so asto match, compliment or content with the image based on a knowledge ofcommonly accepted colour matching rules (which may vary withgeographical region around the world) or with knowledge of the user'spreferences. Such user preference data may be entered by the user in aset up mode, or may be deduced from previous times where a user hasselected a border.

Similarly pattern or ornament of the border may be automaticallyselected based on an analysis of the content of the image and the use ofrules concerning border selection.

Although in the embodiment of the invention just described the entireprocess is performed automatically, in other embodiments the user mayprovide some input to the process. For example, the user may select theparticular generic shape of the image boundary that is to be used. Inaddition, or alternatively, a number of alternative pairings of imageboundary shape and region of interest combination may be presented tothe user for final selection.

FIG. 3 shows the application of the present invention to the generationof a photo-montage. A predefined photo-montage template 14 is shown inwhich a number of image boundaries 15 are located. The image template 14may be entirely predefined, or a user may be able to generate a template14 by controlling one or more template parameters. The template may alsobe created automatically or semi-automatically based on a combination ofthe content of the source images and the user's preferences. Suitabletemplate parameters may include the choice of one or more generic shapesfor the image boundaries, the range of variation of boundary shapes tobe permitted, the minimum and/or maximum number of boundaries to be usedor the distance between individual boundaries. This list is not to beconsidered as being exhaustive. The image boundaries may, for example,be geometric shapes as shown in FIG. 3. In this example, the imageboundaries are all elliptical, however other shapes, for example,rectangles may be used or mixtures of shapes may be used. Also shown aretwo images 16 and 18. In practice a larger number of images will beavailable, the number of images being substantially the same as thenumber of image boundaries 15 included within the photo-montage template14. However, it is not necessary to have exactly the same number ofimages as there are image boundaries 15 within the template 14. Eachimage is analysed in the same manner as that described in relation toFIGS. 1 and 2, and each of the image boundaries 15 making up the montagetemplate 14 is fitted to each analysed image or vice versa In this waythe most appropriate image is matched with the most appropriate imageboundary. For example an ellipse 20 (shown in chain line) may be themost appropriate image boundary for the first image 16. Equally, acircle 22 (shown in chain line) may be the most appropriate imageboundary for the second image 18. The same image may be used more thanonce where the total number of images is less than the total number ofimage outlines 15 in the montage template. In this case, differentregion of interest combinations, as defined by the composition maps, maybe selected such that a number of different image boundaries 15 may befitted to a single image. For example, where an image has two discrete“include” regions of interest, two separate image boundaries may befitted to respective regions and each processed image boundary appliedto the montage. Equally, where the total number of images availableexceeds the number of image boundaries available in the montagetemplates 14, those images having the least satisfactory fit of regionsof interest combination and image boundary 15 will be discarded. Thefitting of images to boundaries may be performed using a crop penaltyanalysis. Thus a crop penalty metric may be obtained for the best cropof each image/hole combination. Any image/hole combinations which cause“include” regions to be cropped or which leave empty space within thehole are penalised. Combinations which cause exclude regions to beincluded may be either automatically discarded, or maintained with avery large penalty—just in case they feature in one of the bestsolutions to the fitting problem. An allocation of images to holes isthen found such that each hole is filled by an image and that differentimages are presented at different holes although if necessary images maybe reused, or some may be excluded. The combination of images and holepairs that minimises the crop penalty is the likely best solution to themontage arrangement. Thus allocation of images to holes is globallyoptimised for the montage.

The user may select some images as “key” images. These images may bepreferentially weighted such that they constitute an increasedproportion of the overall display area. In this way the “key” imageswill tend to be allocated to larger holes in the montage.

In addition to marking some images as key, the user may also be able torank images by order of importance and/or to mark some areas of theimage as “must include”, or to define where some of the images are to beplaced.

In a preferred implementation of the invention the template canautomatically modify itself to ensure that the number of availableimages is matched (or approximately matched to within reasonable limits)to the number of holes.

The aspect ratio of the include regions and the holes in the templateare also approximately matched, although as noted hereinabove thetemplate may further be adjustable to change parameters (such as aspectratio) of the holes defined therein.

FIG. 4 illustrates the generation of a “free form” photo-montage inaccordance with an embodiment of the present invention. A “free form”photo-montage comprises a number of images overlapping each other at theedges, with (generally) no space in between the images. The amount thateach image is overlapped with another image is selected to mask unwantedparts of the image from view. In this embodiment of the invention acentral image 24 is first chosen, the central image being chosen on thegrounds of a large, strongly contrasting subject, for example. Theselection may either be manually performed by the user, or analysis ofthe available images may be performed by the image processor inaccordance with one or more of the previously mentioned compositionrules and a suitable image thus selected.

Regardless of the method of selection, the central image must beanalysed as previously discussed to determine the “include” and“exclude” regions (and by implication “don't care” regions). A furtherimage 26 is then analysed in a similar manner to define the “include”and “exclude” regions for the further image 26. The two images aresubsequently overlapped in such a manner that “exclude” regions of oneof the images are masked from view by a portion of the other image.Subsequent additional images 28, 30, 32 and 34 are analysed and added ina similar manner, further adding to the montage. Thus for a ‘free form’photo-montage the image boundary for each individual image is defined bythe edges of the overlapping images.

If the subsequently added images are to be overlaid on top of existingimages, then the images to be added are preferably cropped along thoseregions that are to cover the existing montage so as to remove any“exclude” regions and optionally some or all of the “don't care” regionsfrom that side. For example, consider that the subsequent image 26 isthe first image to be added to the central image 24. The hatched area 40represents that portion of the central image 24 that constitutes an“exclude” region. However, the small area 42 to the right of thesubsequent image 26 is an “exclude” region of the additional image 26.

This additional area 42 must be cropped from the image 26 prior to thatimage being placed over the central image 24 and overlapping the hatchedarea 40. Thus the “exclude” region of the added image 26 has beenremoved by cropping, and the “exclude” area of the central image 24 hasbeen hidden from view by overlaying the remainder of the added image 26.

The cropping has been illustrated as being rectangular, but this is notnecessarily so. Cropping may occur along a curved path, for examplealong or adjacent the edge of an “include” object, such as a figure ofperson, or along a boundary of the type discussed with reference toFIGS. 1 and 2, i.e. other geometric shapes.

When all of the images have been added to the montage any remaining“exclude” regions at the edge of the montage are removed by cropping.Various additional features may be included in the generation of themontage based on the image analysis. For example, the size of theindividual images comprising the montage may be gradually reduced fromthe centre towards the edge. A further example would be to select imagesfor the montage that generate a gradually darkening overall tone of themontage from the centre towards the edge, or from top to bottom.

In a further embodiment of the invention, a subset of all of theavailable images may be selected as key images. The selection may be onthe basis of the image analysis or may simply be a manual choice by theuser. The most appropriate image boundary for each key image is thenselected as previously described and positioned on the display area in arelatively dominant manner. Image boundaries are then subsequentlydefined for the smaller areas to fill the gaps between the alreadyplaced key image boundaries. The image analysis is preferably not doneon a “one by one” basis but instead is a “global” fit of the availableimages. This approach can be used for both overlapping andnon-overlapping montages.

An alternative embodiment for generating “free form” montages is alsocontemplated. Given a number of images, each image is analysed in thesame manner as described in relation to FIGS. 1 and 2. A number ofgeneric image boundary shapes are available, as previously described,and the best fit between each image and image boundary is generated.

In addition to identifying “include” and “exclude” regions as describedpreviously, in this embodiment farther “don't care” regions are alsoidentified. These are regions of the image that do not strongly conformto the heuristic rules used to identify the “include” and “exclude”regions and are regions that may optionally be included or excluded fromthe image boundary so as to optimise the overall fit of the image, orcombination or regions of interest, to an image boundary.

The image boundaries are then placed on the display area with one ormore portions of one image boundary overlapping at least one other imageboundary. Those regions of an image boundary that are overlapped and arethus obscured from view may only be the remaining previously identified“don't care” regions.

The arrangement of the image boundaries may be entirely automated anddetermined by optimisation of the number of “don't care” regions thatare obscured. Alternatively the user may control the placement of theimage boundaries, although the degree of overlap may still be determinedusing the “don't care” regions.

In the described embodiments of the invention the image analysisrequires the selection of regions of interest from each individualimage.

Various schemes are known for selecting an area of interest from anelectronically captured image. One such known scheme is described in thepresent applicants co-pending UK patent application number 0031423.7entitled “automatic cropping of electronic images”. A brief summary ofthe scheme disclosed in GB0031423.7 will now be described for the sakeof completeness.

FIG. 5 shows an electronic image 50 that shows a girl 52 on a swing, inapproximately the centre of the image. There is also an area to thebottom left of the image in which some flowers 54 are prominent in theforeground. Additionally, part of a small boy and an adult's arm can beseen at the right hand edge 56 of the image. It will be appreciated thatalthough only shown using a greyscale image to aid reproduction, theoriginal image was in colour.

Firstly, the image 50 is processed in such a way that areas of similarcolour, intensity and texture are merged into areas having a uniformcolour and intensity. This can be seen with reference to FIG. 6. Themain parts of the flowers 54 have been merged to areas of uniform colour58, as have the girl's face 64 and trousers 62. Large areas of thebackground have also been merged into areas of uniform colour, forexample the tree 60 on the left hand side of the image. The techniquesfor merging the areas of similar colour are well known. For example, theimage may be converted into a format having a compressed colourvariation and then adjacent areas of similar colour or intensity maythen be merged. Similarly, small areas surrounded by larger areas ofsimilar colour or intensity may also be merged into the larger area.

The resultant image is then further processed by comparing adjacentparts of the image with each other and assigning a rating to each partof the image indicative of the difference in colour and intensitybetween adjacent parts of the image. Using the ratings a further image,referred to as the saliency image, is generated in which the brightnessof an area of the image is proportional to the difference in averagedcolour and intensity of adjacent areas or is a function of the amount bywhich the colour differs from a colour or colours that have been judgedto the background colours. The saliency image for the image picture ofFIG. 5 is shown in FIG. 7. It can be seen that the girl 52 and flowers54 are significantly brighter than other areas of the image. It shouldalso be noted that although the girl is a single object in the realworld, her face and clothing are represented by different values in thesaliency image. If desirable further analysis such as face detection orproximity or similarity grouping can be applied, for example to stopsingle real world objects being identified as many different objects.

Having identified areas of significance within an image one or morecompositional rules are applied to the saliency image to identify the“include” and “exclude” regions. For example, an area of interestlocated at the very edge of an image may be interpreted as a incompletepicture element and would therefore be denoted as an “exclude” region.Other known methods of selecting areas of interest from an image mayequally be applied to the present invention.

FIG. 8 schematically illustrates an image processing system constitutingan embodiment of the present invention. The image processing system 66has two inputs, an image input 68 arranged to receive an image 70 and animage boundary input 72. The image input 68 is connected to one or moreimage sources. In the embodiment shown in FIG. 8, the image input 68 isconnected to an electronic camera 80 and also to a storage device 82,such as a hard disk drive or memory card, that has one or morepreviously captured images electronically stored. The image boundaryinput 72 is arranged to receive one or more of the predetermined imageboundaries, together with the user defined boundaries and definedmontage templates. These are stored in an image boundary memory 74. Thetwo inputs and the image boundary memory are connected to an imageprocessor 76 that performs the image analysis and the fitting of theimage boundary or boundaries to the image 70. The results of theprocessing are passed to an image output 78 arranged to provide theimage bounded by the appropriate image boundary. The image output 78 isconnected to one or more output devices. In the embodiment shown theimage output is connected to a further storage device 84 and a printingdevice 86. The image processing system may be integral with an imagecapture system in which case the framed or montage image may be directlydownloaded to another device such as a printer or computer. Indeed, theimage processing system may also be integral with a printer itself sothat the framed or montage image may be printed immediately. The aboveapparatus can also be used in the generation of free form montages.

It is thus possible to provide an automated method of and apparatus forarranging an image or a plurality of images within a boarder, or amontage.

1. A method of arranging at least one image within a display area, themethod comprising the steps of: analysing the at least one image inaccordance with at least one predefined compositional rule; selecting animage boundary the shape of which, in accordance with said analysis,best fits the at least one image; applying the image boundary to the atleast one image to produce a processed image; and placing the processedimage within said display area, wherein the step of analysing the imagecomprises identifying at least one region of compositional significancebased on color and/or contrast and/or brightness differences withrespect to adjacent regions in the image, and applying at least one of:a) at least one compositional rule to identify regions which should beincluded within the processed image; and b) at least one compositionalrule to identify regions which should be excluded from the processedimage, wherein said step of analysing the image further comprisesidentifying “don't care” regions of images, said “don't care” regionscomprising regions that may be optionally excluded from said imageboundary.
 2. A method of arranging at least one image within a displayarea as claimed in claim 1, wherein the step of analysing the imagecomprises identifying one or more regions of compositional significancebased on color and/or brightness differences with respect to adjacentregions in the image, and applying at least one compositional rule toidentify regions which should be included within the processed image. 3.A method of arranging at least one image as claimed in claim 2, whereinthe image boundary shape placed within said display area equals a numberof selected combinations of regions of interest to be included, saidselection being controlled by at least one heuristic rule.
 4. A methodas claimed in claim 2, wherein the at least one compositional rule isselected from a set comprising a plurality of rules.
 5. A method asclaimed in claim 4, wherein the at least one compositional rule isselected based on its relevance to one or more of the identifiedfeatures.
 6. A method as claimed in claim 2, wherein the step ofanalysing the image comprises identifying one or more regions ofcompositional significance which should be excluded from the processedimage, the regions of computations significance being identified basedon color and/or brightness differences with respect to adjacent regionsin the image.
 7. A method as claimed in claim 6, wherein the imageboundary shape is compared with the image, and those shapes whichinclude the regions which should be included and exclude the regionswhich should be excluded are presented to a user as acceptablesolutions.
 8. A method as claimed in claim 7, in which the imageboundary shape that best includes the features to be included and bestexcludes the features to be excluded is presented to the user as thebest fit.
 9. A method of arranging at least one image within a displayarea as claimed in claim 1, wherein the step of analysing the imagecomprises identifying one or more regions of compositional significancebased on color and/or brightness differences with respect to adjacentregions in the image, and applying at least one compositional rule toidentify regions which should be excluded from the processed image. 10.A method as claimed in claim 9, wherein the at least one compositionalrule is selected from a set comprising a plurality of compositionalrules on the basis of its relevance to one or more of the identifiedfeatures.
 11. A method of arranging at least one image as claimed inclaim 9, wherein one or more parameters of said shape of image boundaryis adjusted to optimise said best fit.
 12. A method of arranging atleast one image as claimed in claim 9, wherein said image boundary shapeis selected from a plurality of geometric shapes.
 13. A method ofarranging at least one image as claimed in claim 9, wherein said displayarea comprises a predefined arrangement of image boundaries.
 14. Amethod of arranging at least one image as claimed in claim 9, wherein atleast one of the number, position, size, shape and aspect ratio of theat least one image boundary is defined by a user.
 15. A method ofarranging at least one image as claimed in claim 9, wherein at least oneparameter of the style of the display area is controlled by a user. 16.A method of arranging at least one image as claimed in claim 1, whereina plurality of images are arranged and the number of image boundariesplaced within said display area is equal to the number of images.
 17. Amethod of arranging at least one image as claimed in claim 1, wherein aplurality of image boundaries are placed within said display area, atleast some of the image boundaries overlapping at least a portion of oneother of said image boundaries.
 18. A method of arranging at least oneimage as claimed in claim 1, wherein said overlapping portions comprisesaid “don't care” regions.
 19. A method of arranging at least one imageas claimed in claim 1, wherein a plurality of images are arranged andthe placement of the images is controlled by one or more parameters ofthe images.
 20. A computer-readable medium storing a program productcomprising computer readable instructions for causing a data processorto execute the method as claimed in claim
 1. 21. A method of arranging aplurality of images within a display area, the method comprising thesteps of: analysing each of said plurality of images in accordance withat least one compositional rule to thereby identify one or more regionsof each image to be excluded from said image; placing said analysedimages within said display area, such that some images overlap a portionof at least one other image, whereby said overlapping portions obscureone or more of said regions to be excluded, wherein said step ofanalysing an image comprises: identifying one or more features ofcompositional significance based on color and/or brightness differenceswith respect to adjacent regions in the image; identifying a number ofregions of interest; selecting at least one combination of one or moreof said regions of interest; applying said at least one composition ruleto those said features of compositional significance located within saidselected combination of regions of interest to thereby identify “mustinclude” and “must exclude” regions of said selected combination; andidentifying “don't care” regions of images, said “don't care” regionscomprising regions that may be optionally excluded from an imageboundary.
 22. A method of arranging a number of images as claimed inclaim 21, wherein regions to be excluded for each image that are notobscured by said overlapping portions are cropped from said image.
 23. Amethod of arranging a plurality of images within a display area asclaimed in claim 21, wherein the method further comprises identifying“don't care” regions which may be wholly or partially overlapped byanother image.
 24. A method of arranging a plurality of image accordingto claim 21, wherein placement of said images is controlled by one ormore parameters of said images.
 25. A computer-readable medium storing aprogram product comprising computer readable instructions for causing aprogrammable data processor to execute the method as claimed in claim21.
 26. An image processing system for arranging a number of imageswithin a display area, comprising: an input arranged to receive saidimages; an image processor coupled to said input; and an output coupledto said image processor, and arranged to output said arrangement ofimages within the display area; wherein the image processor is arranged:to analyse at least one image in accordance with at least one predefinedcompositional rule; to select an image boundary the shape of which, inaccordance with said analysis, best fits said image; to apply the imageboundary to the at least one image to produce a processed image; and toplace the processed image within said display areas, wherein the step ofanalysing the image comprises identifying at least one region ofcompositional significance based on color and/or brightness differenceswith respect to adjacent regions in the image, and applying at least oneof: a) at least one compositional rule to identify regions which shouldbe included within the processed image; and b) at least onecompositional rule to identify regions which should be excluded from theprocessed image, wherein said step of analysing the image furthercomprises identifying “don't care” regions of images, said “don't care”regions comprising regions that may be optionally excluded from saidimage boundary.